Empowering Adaptive Early-Exit Inference with Latency Awareness

With the capability of trading accuracy for latency on-the-fly, technique adaptive early-exit inference has emerged as a promising line research to accelerate deep learning inference. However, studies in this commonly use group thresholds control accuracy-latency trade-off, where thorough and general methodology on how determine these not been conducted yet, especially with regard common requirements average latency. To address issue enable latency-aware inference, present paper, we approximately formulate threshold determination problem finding accuracy-maximum setting that meets given requirement, then propose method tackle our formulated non-convex problem. Theoretically, prove that, certain parameter settings, finds an approximate stationary point Empirically, top various models across multiple datasets (CIFAR-10, CIFAR-100, ImageNet two time-series datasets), show can well handle requirements, consistently good settings negligible time.